MEMORANDUM May 27, 2020 


TO: Montra Rogers, Ed.D. 
Officer, Secondary Curriculum and Development 


FROM: Allison Matney, Ed.D. 
Officer, Research and Accountability 


SUBJECT: A QUASI-EXPERIMENTAL STUDY ON THE USE OF SECONDARY TEACHER 
DEVELOPMENT SPECIALISTS TO IMPROVE TEACHERS’ INSTRUCTIONAL 
COMPETENCIES AND STUDENT ACHIEVEMENT, 2019-2020 


This program evaluation examined the perceptions of secondary Teacher Development 
Specialists (TDS) regarding their coaching competencies in key content areas along with the 
perceptions of teachers regarding the impact of secondary TDS on their instructional practices 
and student achievement. Propensity score matching allowed for measurement of intensive 
treatment effects (at least 80% to 90% of time spent coaching) using District-level Assessment 
(DLA) data. DLAs are aligned to STAAR Blueprints and the Texas Essential Knowledge and 
Skills (TEKS). 


Key findings include: 

e The study samples were comprised of 53 secondary TDS and 254 teachers coached by 
secondary TDS at 41 struggling schools in HISD. 

e Teachers, overwhelmingly, expressed that secondary TDS displayed expertise and 
credibility, listening, objectivity, and clear and effective communication skills as coaches. 

e Teachers consistently noted that secondary TDS were knowledgeable about curriculum and 
instruction, pushed them to change and expand their practices, and helped them meet the 
individual needs of their students. 

e When controlling for students’ background characteristics and eighth-grade performance on 
content-related STAAR tests, the effects of intensive secondary TDS treatment led to mostly 
comparable or higher performance of students in schools with large proportions of at-risk 
students compared to schools with considerably less at-risk students on Algebra 1, Biology, 
English language arts (ELA), and U.S. History DLAs. 

e Considering the alignment of DLAs with STAAR, intensive secondary TDS treatment 
resulted in mean passing rates that advanced closer to the STAAR Meets Grade level 
standard in the content areas measured in this evaluation, reflecting academic benefits for 
students in struggling schools. 


Further distribution of this report is at your discretion. Should you have any further questions, 
please contact me at 713-556-6700. 
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A Quasi-experimental Study on the Use of Secondary Teacher Development Specialists to Improve 
Teachers’ Instructional Competencies and Student Achievement, 2019-2020 


Prepared by Venita R. Holmes, Dr.P.H. 


Abstract 

This program evaluation examined the perceptions of secondary Teacher Development Specialists (TDS) regarding 
their coaching competencies in English language arts (ELA), mathematics, science, social studies, literacy, and 
instructional technology. Moreover, the study assessed the extent secondary TDS’ coaching practices improved 
teachers’ instructional competencies to support student achievement. The study samples were comprised of 53 TDS 
and 254 teachers at 41 struggling schools in the Houston Independent School District (HISD). Intensive coaching 
was provided, with at least 80% to 90% of TDS’ time spent delivering coaching services in the targeted schools 
during the 2019-2020 academic year. Pearson correlations revealed a strong positive association between TDS’ 
listening behavior and their ability to be objective and provide clear and effective communication to teachers. 
Teachers, overwhelmingly, expressed that TDS displayed expertise and credibility, listening, objectivity, and clear 
and effective communication skills as coaches. Moreover, teachers consistently noted that TDS were knowledgeable 
about curriculum and instruction, pushed them to change and expand their practices, and helped them meet the 
individual needs of their students. Propensity score matching allowed for measurement of intensive TDS treatment 
effects at targeted schools using non-randomized District-level Assessment (DLA) data, while controlling for students’ 
background characteristics (economic status, gender, at risk, gifted/talented, special education, and eighth-grade 
performance on content-related State Assessment of Academic Readiness (STAAR)) tests. The effects of intensive 
TDS treatment led to mostly comparable or higher performance of students in schools with large proportions of 
at-risk students compared to schools with considerably less at-risk students based on students’ fall DLA results in 
Algebra 1, Biology, ELA, and U.S. History, with passing rates advancing closer to the STAAR Meets Grade level 
standard. Consequently, intensive TDS treatment appeared to have benefits for students in struggling schools. 


Background 

High-quality teacher professional development is 
widely-recognized as an important factor to facilitate 
student learning and help students attain their academ- 
ic goals (Addimando, 2019; Darling-Hammond, 2000; 
Wray, Medwell, Fox, & Poulson, 2000). “Teacher coach- 
ing has emerged as a promising alternative to traditional 
models of professional development” (Kraft, Blazar, & 
Hogan, 2017, p. 1). Teacher coaches function as experts 
to novice teachers (Poglinco et al., 2003) and help them 
think critically about their practices to benefit all stu- 
dents, including students with special needs (Hasbrouck 
& Denton, 2007). With emphasis on content and peda- 
gogy-building, teacher coaches foster effective learning 
communities, whole-school reform, leadership devel- 
opment, and improvements in resource utilization in 
schools (Hershfeldt et al., 2012) (Figure 1). Student-cen- 
tered models allow coaches and teachers to work in part- 
nership to set learning targets and work collaboratively to Figure 1: ELA TDS modeling a Read Aloud Think Aloud for 
meet those targets (Sweeney, 2013). teachers during early release professional development 


THEORY OF ACTION: Secondary Teacher Development Specialists spending at least 80% to 90% of their time providing direct, peer, instructional coaching to content 
teachers will yield improvements in teacher competencies, instructional practices, and student achievement. 


PROFESSIONAL DEVELOPMENT - COACHING 


Intermediate Outcomes 


Long-term Impact 


* Individualized — Coaching consists of face-to-face, one-on-one sessions. 


. Intensive — Coaches and teachers interact weekly at schools in greatest 
need of support and periodically at other schools on an ad hoc basis, with 
80%-90% of the coaches’ time spent working directly with teachers on 
campuses. 


. Sustained — Teachers receive on-going coaching support throughout the 


academic year. 


. Context-specific — Teachers are coached on practices within the context 
of their own classroom, which is guided by their Professional Learning 
Communities (PLCs) and modeled through demonstration lessons. 


TEACHER KNOWLEDGE STUDENT OUTCOMES 


Students increase academic 


Teachers build knowledge in the core content y 
achievement. 


areas, literacy, and instructional technology. 


Teachers build pedagogical knowledge on Students increase attendance. 


how to teach students of various cultures and 


i ‘ . Decrease student disciplinary 
socio-economic backgrounds, and supporting 


actions that result in out-of-school 


diverse learners. time. 


° Focused — Coaches work with teachers to differentiate instruction and to 
develop evidence-based strategies for targeted subgroups of the popu- 
lation, including English learners, students with disabilities, gifted and 


talented, and at-risk students. 


CURRICULAR MATERIALS 


District Master Courses 

Literacy Routines (instructional strategies to build literacy scaffolds to allow En- 
glish learners to access grade-level content in English) 

Literacy Empowered (high school) 

Literacy in the Middle (middle school) 

Reading and Writing Training (for teachers and leaders) 

Planning Instruction with Mastery in Mind Protocol 


THACHER BEHAVIOR. TEACHER BEHAVIOR 
Teachers focus on instructional practices to 
deliver consistent and effective classroom 
management strategies. 


Increase Teacher Appraisal and 
Development System (TADS) 
Instructional Practices Rating. 
Teachers are better able to identify teaching 
strategies to support student learning and to 
meet targeted student outcomes. 


Increase Teacher Retention. 


Figure 2: HISD Secondary Teacher Development Specialists Theory of Action, HISD Secondary Curriculum and Development, 2019-2020 


Over the years, Teacher Development Specialists (TDS), at 
the secondary level, have been used as a coaching model to im- 
prove teacher quality, teacher retention, and student achievement 
in the Houston Independent School District (HISD). While most 
coaching models focus on single targets for change (Dufrene et 
al., 2014; Dudek, Reddy, Lekwa, Hua, & Fabiano 2019), the HISD 
TDS model addresses multiple areas where teachers may struggle. 
In alignment with Kraft, Blazar, and Hogan (2017), the secondary 
TDS theory of action incorporates individualized, intensive, sus- 
tained, context-specific, and focused coaching along with instruc- 
tional strategies and curricular materials to support literacy and 
attainment of content-knowledge (Figure 2). 

For the past two years, Secondary Curriculum and Develop- 
ment TDS have engaged in the work of Sweeney (2013) to actu- 
alize a student-centered coaching model. The TDS is not charged 
with “fixing” the teacher; they work collectively to ensure student 
success. Teachers improve their practice because the focus is in- 
tentionally on students’ actions and outcomes. 

Competencies are the skills and knowledge that enable 
teachers to be successful (Jackson, 1990). The best teachers are 
proficient in instructional delivery, formative assessment, and 
classroom management (Hattie, 2009) (Figure 3 through Figure 
5). Few studies have focused on coaches’ perceptions of their com- 
petencies to work with teachers and teachers’ perceptions of how 
coaches impacted their instructional practices to increase student 
achievement. Moreover, few studies have sought to link student 
achievement with the implementation of intensive teacher coach- 
ing, where coaches spent at least 80% to 90% of their time in strug- 
gling schools (Desimone, 2009). To that end, this study examined 
gaps in the research by addressing the following questions. 


Research Questions: 
1. To what extent did secondary Teacher Development Specialists 
perceive their coaching as beneficial toward improving teachers’ 


instructional competencies to support student achievement? 

2. What were teachers’ perceptions regarding the impact of 
secondary TDS’ coaching practices toward improving their 
instructional skills and increasing student achievement? 

3. What was the effect of the intensive secondary TDS coaching 
model on student performance in mathematics, science, reading, 
and social studies at targeted schools? 


Limitations 

There were several limitations to the study. Secondary TDS 
and teacher participation in surveys was voluntary. This may have 
resulted in selection bias due to the underrepresentation of edu- 
cators in specific content areas. To mitigate this limitation, the 
survey time was extended and multiple attempts were made to 
communicate the intent of the survey and deliver the survey links 
to targeted participants. In addition, low student participation on 
District-level Assessments (DLAs) was evident in the data, and 


Figure 3: Social studies TDS leading PLC at Sugar Grove Academy 


Figure 4: Math TDS leading teachers through the Planning Instruction 
with Mastery in Mind protocol during Achieve 180 PD 


participation in DLAs was not randomized. To mitigate these lim- 
itations, propensity-score matching techniques were used to select 
students who completed the assessments to generate treatment and 
control groups from two distinct study populations, controlling for 
previous performance on aligned content area tests and students’ 
background characteristics. 


Review of the Literature 

The research has shown that teacher professional 
development has the potential “to empower educators and 
communities of educators to make complex decisions; identify and 
solve problems, connect theory to practice, and improve student 
outcomes” (Katamei & Omwono, 2015, p. 112). A study conducted 
by The New Teacher Project found that school districts spend an 
average of nearly $18,000 per teacher, per year, or about six to nine 
percent of their annual budgets on resources, including time and 
money toward training, mentoring, evaluating, and ongoing job- 
embedded experiences to help teachers improve. The challenge for 
many educational institutions is to identify how best to support 
teachers using the most cost-effective methods (Quintero, 2019). 

A variety of professional development methods have been 
used to support teacher quality. Specifically, traditional methods 
employ experts to deliver face-to-face or online professional de- 
velopment (Limongelli, Sciarrone, Temperini, & Vaste, 2011; 
Zhang, Liu, & Wang, 2017). While traditional approaches have 
the potential to adapt to teachers’ learning styles, teachers may 
not consistently benefit in acquiring high-level skills and applying 
pedagogies in practice (Limongelli, Sciarrone, Temperini, & Vaste, 
2011). Workshop-style professional development models are lim- 
ited in helping teachers learn, implement, and receive support 
during implementation of new skills (Carlisle & Berebitsky, 2011; 


Figure 5: Secondary science TDS creating high-quality assessments 


Gulamhussein, 2013). They assume that the difficulty in teachers’ 
learning is due to their lack of knowledge rather than transferring 
knowledge and implementing the skill in the classroom (Dudek et 
al., 2019). Student-centered models have promise toward ensuring 
that students have met all academic standards by framing coach- 
ing conversations on students’ learning targets, conducting regular 
analysis of students’ work, and providing evidence of students’ 
learning, while co-planning and co-delivering instruction (Swee- 
ney, 2013). The process is on-going, involving individual teachers, 
teams of teachers, and school leaders (Sweeney, 2013). 

Personalized learning, such as peer coaching and instruction- 
al coaching models, has become increasingly prevalent to develop 
teachers’ expertise in content areas, strategies, and to address their 
individual learning needs. Coaches provide assistance, feedback, 
and support, and share their expertise to enhance learning (Ack- 
land, 1991; Kohler, Crilley, Shearer, & Good, 1997). 

Kraft, Blazar, and Hogan’s (2017) meta-analysis of 60 
instructional coaching evaluations found large, positive effects on 
student achievement. Gregory et al. (2016) found more engagement 
among minority students using coaching. Ma, Xin, and Du (2018) 
conducted a quasi-experimental design study involving 20 
Mandarin teachers to determine the effectiveness of a personalized 
learning approach that consisted of peer coaching. The study found 
that the peer coaching-based personalized learning approach had a 
positive effect on teachers’ participation in learning opportunities, 
learning design skills, and in-practice teaching abilities. 

Abbasian and Matin (2018) conducted a study using teachers 
whose experience ranged from ten to fifteen years, along with 
a experimental group of 156 students and a control group of 
151 students. The study found a significant change in students’ 
achievement test scores due to peer coaching. Other studies posted 
similar findings relative to the positive effects of peer coaching 
on student achievement (Prince, Snowden, & Matthews, 2010; 
Richards, 2003; Sunderman & Kim, 2007). 


Methods 
Study Population and Sample 

There were two study populations in this program evaluation. 
The first study population consisted of approximately 75 secondary 
TDS who coached teachers in ELA, mathematics, science, social 
studies, instructional technology, and literacy during the 2019-2020 
academic year. Literacy and technology were included considering 
that teachers incorporated related practices when coaching in the 
content areas. The second study population was comprised of 
approximately 425 teachers who were coached by secondary TDS 
over the same time frame. These groups were selected because 
teachers were direct recipients of TDS coaching, and could provide 
reliable data on TDS impact. The identities of TDS and teachers 
coached by TDS were provided by Secondary Curriculum and 
Development administrators. These study populations were used 
to generate a TDS sample and a teacher sample based on survey 
participation. 

A total of 54 secondary TDS completed a survey administered 
using Google Forms. An inspection of the data led to the removal 
of one respondent due to extreme values that could seriously 
influence the analyses. Consequently, survey data for 53 secondary 
TDS were used in the study, yielding a response rate of 72%. A 
total of 254 teachers completed the teacher survey, resulting in a 
response rate of 60%. Teachers at 19 HISD schools located in low- 
income, high-risk school communities were targeted for the study. 
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However, teachers who completed surveys indicated that they 
taught at 41 schools. (See Appendix A on p. 12 for background 
characteristics of the study samples.) 


Data Collection/Data Analyses 

Surveys administered to secondary TDS and teachers coached 
by the TDS were adapted from Coaching for Change: Tools 
for Assessing Instructional Coaching Competencies (Insight 
Education Group, 2011). The questions on both surveys were 
aligned. The TDS survey consisted of four indicators: Credibility, 
Listening, Objectivity, and Clear and Effective Communication. 
The indicators measured in the teacher survey were Expertise 
and Credibility, Listening, Objectivity, Clear and Effective 
Communication, and Overall Impact. (See Appendix B on p. 13 
for definition of terms.) 

The TDS survey allowed respondents to self-reflect on 
their confidence as coaches to improve teachers’ instructional 
competencies and student achievement. Additional information 
was captured on the TDS survey, including their content area 
and years of teaching experience. Open-ended questions allowed 
TDS to document challenges in providing TDS support and 
to indicate what helped them to be successful as a TDS. The 
teacher survey measured their perceptions of secondary TDS’ 
coaching practices toward improving their instructional skills and 
student achievement. Teachers were asked open-ended questions, 
including what challenges they encountered, if any, working with 
secondary TDS as a coach and how working with secondary TDS 
as a coach helped them to be more successful as an educator. 

Descriptive statistics were calculated to determine the level 
of confidence perceived by secondary TDS on survey items. Using 
a Likert-type scale, a coding system was established: “completely 
confident” = “4”, “fairly confident” = “3”, “somewhat confident” 
= “2”, “slightly confident” = “1”, and “not at all confident” = “0”. A 
likert scale measuring agreement was used on the teacher survey: 
“strongly agree” = “4”, “agree” = “3”, “disagree” = “2”, “strongly 
disagree” = “1”, and “N/A” (not applicable). The percentage 
of TDS and teachers who rated the items on each indicator, the 
means, and standard deviations were presented in this evaluation. 
Missing and not applicable data were not included in calculations. 

Pearson’s correlation (r) was used to examine the relationship 
among the indicators. According to Cohen (1988), the effect size is 
lowifthe value ofrvariesaround0.1,mediumifrvariesaround0.3,and 
large ifr varies more than 0.5. Non-parametric statistics (Kruskall- 
Wallis test) assessed differences in TDS and teachers’ confidence 
across years of experience at four levels (“less than | year”, “1 up 
to 2 years”, “2 up to3 years”, and “more than 3 years”) and across 
content areas. The level of statistical significance was p < .05. 

Fall 2019 HISD DLA (Part 1, multiple choice) ELA, Algebra 
I, Biology, and U.S. History results were used to determine the 
impact of the intensive secondary TDS model (at least 80% to 
90% of the time coaching) on student achievement. DLAs were 
used as an alternative academic performance measure considering 
that the spring 2020 STAAR tests were not administered to 
students due to the coronavirus pandemic. Moreover, literacy 
and instructional technology were not measured considering 
that there were no associated DLAs. DLAs were developed 
by HISD curriculum experts. DLA test items were aligned to 
the Texas Essential Knowledge and Skills (TEKS) and STAAR 
Blueprints. DLAs helped HISD administrators determine how 
well students mastered the curriculum at the point in time the 


Table 1: Type of power analysis: A priori - compute required sample size 


for independent samples 


Input Parameters Output Parameters 

Effect size d 0.5 Noncentrality 3.623 
parameter 

a error prob (Tupe I 0.05 Critical t 1.971 

error rate 

Power (1 - B err prob). 0.95 Df 208 
Sample size group | 105 
Sample size group 2 105 
Total sample size 210 


assessment was administered. While low participations rates are 
common at the high-school level, a study conducted in 2014-2015 
found a large, positive correlation between DLAs and STAAR 
End-of-Course (EOC) performance in all subjects (Houston 
Independent School District, Student Assessment Department 
(personal communication, 1/8/2020)). An alignment of DLAs were 
discussed based on STAAR performance categories (Approaches 
Grade Level, Meets Grade Level, and Masters Grade Level). 
Calculations related to STAAR were based on the group mean 
DLA score divided by the maximum number of items for the test. 

IBM Cognos was used to extract students’ DLA scores. 
G*Power software was used to estimate the sample size needed prior 
to the study to detect a statistically significant difference in students’ 
test performance, considering that a difference truly existed (Faul, 
Erdfelder, Lang, & Buchner, 2007; Pitner, Yu, & Brown, 2013). For 
a two-tailed independent t-test, G*Power determined that a sample 
size of 105 was needed for the treatment and control groups, with 
a critical t-value at 1.971, a significance factor of 0.05, beta value 
equal to 0.5, and degrees of freedom equal to 208 (Table 1). An 
effect size (d) of .5 is considered medium effect (Cohen, 1988). 
There was a 95% chance of correctly rejecting the null hypothesis 
that there was no difference in the academic performance of students 
at schools with intensive secondary TDS and schools without 
intensive secondary TDS. While the student samples varied for each 
test, there was more than sufficient data to conduct the analyses. 

Propensity score matching was conducted using SPSS 
software to adjust for treatment effects. Propensity score 
matching is considered an alternative to the commonly used 
regression adjustment (Stuart, 2010). For this analysis, nearest- 
neighbor matching with replacement was used. The logic behind 
propensity score methods is that balance on observed covariates 
is achieved through careful matching on a single score — the 
estimated propensity of selecting the treatment (Stuart, 2010). 
The propensity score is defined as the probability of receiving 
treatment based on measured covariates. Covariates used for 
matching in this evaluation were gender, economic status, at risk, 
gifted/talented, and special education, along with students’ eighth- 
grade STAAR reading (ELA DLA), math (Algebra I DLA), science 
(Biology DLA), and social studies (U.S. History DLA) scores. 
The quality of matches can be affected by the order in which 
subjects were selected for matching and the maximum permitted 
difference between matched subjects (the “caliper’’) (Lunt, 2014). 
The caliper used in the analyses was .25 standard deviations 
considering that Rosenbaum and Rubin (1985) used this caliper 
based on Cochran and Rubin’s (1973) logistic regression model 
to predict exposure. Matches were selected with replacement to 
improve imbalance between the groups, and in a randomized order. 
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Results 


To what extent did secondary Teacher Development 
Specialists perceive their coaching as beneficial toward 
improving teachers’ instructional competencies to support 
student achievement? 


TDS’ confidence to improve teachers’ instructional 
competencies to support student achievement was based on 
their responses to survey items on four indicators: (1) Expertise 
and Credibility, (2) Listening, (3) Objectivity, and (4) Clear and 
Effective Communication. The findings are depicted in Table 2 
through Table 5 (Appendix C, pp. 14-15). Results depict the 
percentages of TDS who indicated that they were completely 
confident”, “fairly confident”, “somewhat confident”, “slightly 
confident”, and “not at all confident”. Descriptive statistics, 
including mean confidence scores and standard deviations 
(S.D.) were included. 

On the Expertise and Credibility indicator (Table 2, 
Appendix C, p. 14), the majority of respondents perceived 
that they were “completely confident” on all survey items. The 
findings ranged from 54.7% (I have connected with my coaching 
mentor to get support in my current position) to 83.0% (I share 
what I have learned in professional development with my staff). 
Expertise and Credibility confidence ratings ranged from 2.85 
out of 4 (“less than fairly confident”) to 3.77 out of 4 (“more 
than fairly confident”) (Table 2a). TDS perceived themselves as 
“more than fairly confident” on five out of eight items (62.5%). 

Table 3 provides TDS perceptions on the Listening 
indicator (Appendix C, p. 14). The majority of TDS respondents 
perceived that they were “completely confident” on all items, 
except one (90.9%). The findings ranged from 49.1% (I 
effectively use facilitative processes that draw out reluctant 
participants) to 83.0% (I give teacher and staff concerns and 
ideas serious consideration). Listening confidence ratings are 
depicted in Table 3a. The ratings ranged from 3.28 (“less than 
slightly confident”) to 3.83 (“more than fairly confident”). TDS 
perceived themselves as “more than fairly confident” on eight 
out of eleven items (72.7%). 

TDS perceptions on the Objectivity indicator are depicted in 
Table 4 (Appendix C, p. 15). The majority of TDS respondents 
perceived that they were “completely confident” on eight out 
of nine items (88.9%). The findings ranged from 41.5% (I use 
a variety of techniques to challenge staff to acknowledge their 
own assumptions) to 77.7% (I ask clarifying questions to draw 
out details and help teachers synthesize information). Objectivity 
confidence ratings are depicted in Table 4a (p. 6). The ratings 
ranged from 3.15 (“less than slightly confident”) to 3.75 (“more 
than fairly confident”). Overall, TDS perceived themselves as 
“more than fairly confident” on six out of nine items (66.7%). 

Table 5 illustrates TDS’ perceptions on the Clear and 
Effective Communication indicator (Appendix C, p. 15). The 
majority of respondents perceived that they were “completely 


Table 2a: TDS Expertise and Credibility confidence ratings, spring 2020 


Confidence 
Expertise and Credibility Indicator (n = 53) Ratings S.D. 
mn 0 
More than Fairly Confident 
I share what I have learned in professional development 3.77 0.542 
with my staff. 
I bring my own knowledge to bear in conversations while 3.75 0.515 
acknowledging and integrating the knowledge of others. 
I am aware of the classroom realities at my site, and adapt 3.70 0.607 
my coaching accordingly. 
I have the credibility to help teachers reflect deeply on 3.62 0.627 
their own practices. 
I guide teachers and other colleagues to-wards relevant 3.60 0.716 


and meaningful professional development activities. 


Less than Slightly Confident 


The teachers I work with find my experience relevant to 3.49 0.775 
their own needs. 


I regularly observe in classrooms at my own site and at 3.32 1.173 
other sites so that I can identify instructional patterns and 
best practices. 


Ihave connected with my coaching mentor to get support 2.85 1.537 
in my current position. 


Table 3a: TDS Listening confidence ratings, spring 2020 


Confidence 
Listening Indicator (n = 53) Ratings S.D. 
u o 
More than Fairly Confident 
I give teacher and staff concerns and ideas serious consider- 3.83 0.379 
ation. 
When I have conversations with teachers, I avoid distractions 3377 0.466 
and devote my full attention to their needs at that particular 
time. 
lask clarifying questions when appropriate to ensure that I 3:75 0.515 
understand what teachers are saying. 
I provide safe opportunities for teachers and staff to reflect 3.75 0.477 
honestly on their practice. 
I can hear and process an issue from multiple perspectives 3.72 0.568 
(i.e., teacher, student, administrator, etc.). 
I regularly provide a forum for teachers to express concerns 3.68 0.673 
and ask questions. 
I effectively listen to and synthesize what teachers and staff 3.68 0.547 
Say. 
Iam able to distinguish between concerns that simply need 3.64 0.591 
validation and those that require action. 
Less than Slightly Confident 
I provide opportunities for teachers to dialogue with each 3.47 0.799 
other, not just with me. 
I can ensure that all participants are heard by their colleagues 3.42 0.719 
during group processes. 
l effective use facilitative processes that draw out 3.28 863 
reluctant paticipants 


confident” on 100% of the twelve survey items. The findings 
ranged from 64.2% (I model the process of articulating goals that 
will elicit specific outcomes) to 84.9% (I am open to feedback on 
my communication style and am willing to change my practices 
in order to meet the needs of teachers and staff). Table 5a (p. 6) 
shows confidence ratings ranged from 3.57 to 3.85 (“more than 
fairly confident”). Consequently, TDS perceived themselves as 


“more than fairly confident” on each of the twelve items (100.0%). 

Correlation analyses examined the strength of the relationships 
between the indicators used in this evaluation (Table 6a, p. 6). 
There were large, positive associations among the indicators, with 
the largest association found between TDS’ perceptions of their 
listening practices and objectivity when coaching teachers, r = 
.872, p< .001 (Table 6a). The second largest association was found 
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Table 4a: TDS Objectivity confidence ratings, spring 2020 


Confidence 
Objectivity Indicator (n = 53) Score S.D. 
ul o 
More than Fairly Confident 
I balance feedback that I give to teachers and staff between 3.75: 0.434 
positive and constructive. 
L ask clarifying questions to draw out details and help teach- 3.72 0.568 
ers synthesize information. 
I can listen to my collaborating teacher’s plans and observe 3.55 0.607 
his/her actions without judgment, analysis, or critique. 
I make sure that I have all the relevant information about a 3.55 0.539 
situation before. 
I actively solicit input from multiple stakeholders before 3.53 0.823 
taking action on an issue. 
When facilitating a controversial conversation between more 3.53 0.608 


than one party, I am able to remain neutral even when my 
personal beliefs might place me on one side of the dialogue. 


Less than Slightly Confident 


I am able to dialogue with teachers without projecting my 3.45 0.774 
own values and beliefs on the conversation. 

I am effective in pushing staff members to change/expand 3.42 0.77 
their practices while simultaneously acknowledging their 

successes. 

I use a variety of techniques to challenge staff to acknowledge 3.15 0.949 


their own assumptions. 


among TDS’ perceptions of their listening practices and their 
ability to deliver clear and effective communication to teachers, r 
= .870, p< .001. These findings suggest that the more confidence 
TDS perceived in their listening behavior, the more objectivity 
displayed in their coaching relationships with teachers. Moreover, 
the more objectivity displayed in coaching, the clearer and more 
effective communication perceived by TDS in their coaching 
practices. 

The Kruskal-Willis test was conducted to determine whether 
there was a statistically significant difference in TDS’ perceptions 
on the indicators, taking into account years as a TDS (Table 6b). 
The test revealed no statistically significant differences in years 
as a secondary TDS and TDS’ perceptions on Credibility (H(3) 
= .979, p = .806); Listening (H(3) = 3.508, p = .320); Objectivity 
(H(3) = 5.769, p = .123); and Clear and Effective Communication 
indicators (H(3) = .906, p = .824) (Table 6b). However, the mean 
rank was highest on the Objectivity indicator among TDS with 2 
to 3 years (Mean Rank = 32.6) and | to 2 years of experience 
(Mean Rank = 31.5), meaning that these TDS groups perceived 
themselves as more confident coaching teachers relative to these 
indicators. 

The Kruskal-Wallis test was also conducted to determine 
whether there was a difference in TDS’ perceptions on indicators 
based on the content area (Table 6c, p. 7). The test found no 
statistically significant differences relative to Credibility (H(3) 
= 6.68, p = .246); Listening (H(3) = 8.94, p = .111); Objectivity 
(H(3) = 8.51, p = .130); and Clear and Effective Communication 
indicators (H(3) = 2.88, p = .719). However, the mean rank was 
highest on the Credibility indicator among TDS who coached in 
math and social studies; highest on the Listening indicator among 
TDS who coached in science and social studies; highest on the 
Objectivity indicator among TDS who coached in science and 
instructional technology; and highest on the Clear and Effective 
Communication indicator among TDS who coached in science and 
social studies. 


Table 5a: TDS Clear and Effective Communication confidence ratings, spring 2020 


Confidence 
Clear and Effective Communication Indicator (n = 53) Score S.D. 
u oO 
More than Fairly Confident 
Iam open to feedback on my communication style and am 3.85 0.361 
willing to change my practices in order to meet the needs of 
teachers and staff. 
I keep professional conversations with teachers and staff 3.83 0.379 
focused on the achievement of agreed-upon outcomes. 
During conversations with teachers, I deliberately choose to 3.81 0.395 
use language that is clear, non- threatening, and objective. 
I use communication formats that are convenient for teach- 3.77 0.466 
ers and staff (i.e., bulletin boards, email, paper memos). 
I disseminate written information in a timely fashion. 3.77 0.423 
I return phone calls and emails in a timely fashion (within 24 3.74 0.486 
hours, when possible). 
When appropriate, I share agendas and planning tools with 3.72 0.533 
teachers and staff prior to meetings. 
I provide clear, specific, and accurate feedback to teachers 3.72 0.495 
about classroom observations. 
I model ways to communicate effectively. 3.72. 0.495 
I provide clear processes and agendas for meetings and 3.68 0.51 
group conversations. 
I meet face to face with teachers in a timely fashion follow- 3.62 0.686 
ing classroom observations to debrief the observation and 
clarify written feedback. 
I model the process of articulating goals that will elicit 3.37 0.665 
specific outcomes. 


Table 6a: Pearson product-moment correlation between measures of per- 


ceived coaching confidence 


Scale 1 2 3 4 
1. Credibility 2 


2. Listening 626** - 

3. Objectivity 586** = .872** - 

4. Clear and Effective Communication .631** — .870** —.820** - 
Mean 3.514 3.636 3.516 3.733 
S.D. 4739 4162 = .5022—.3311 


p< 001 (2-tailed) (N = 53) 


Table 6b: Non-parametric Kruskal-Willis test comparing years as a TDS 


with confidence indicators, spring 2020 


Mean Rank 

Less 1lto2 2to3 More Chi- Pp 
Indicator thanl years years than3 square 

year years 
1. Credibility 23.3 28.8 26.8 28.1 979 806 
2. Listening 20.0 29.7 26.9 29.5 3.508 320 
3. Objectivity 18.6 31.5 32.6 27.1 5.769 123 
4. Clear and Effective 24.0 273 30.5 27.2 .906 824 
Communication 
N 12 12 8 21 


Table 6c: Non-parametric Kruskal-Willis test comparing TDS by content areas and perceptions on confidence indicators, spring 2020 


Mean Rank 
ELA Instructional 
Technology 

Indicator 

Credibility 22.2 20.3 
Listening 27.8 15.9 
Objectivity 24.5 32.4 

Clear and Effective Communication 25.1 26.1 

N 17 4 


Literacy Math Science Social Chi- p 
Studies square 
22.8 34.9 29.3 34.2 6.68 .246 
15.2 25:3 33.9 315 8.94 mane 
21.8 17.4 35.8 28.8 8.51 .130 
24.0 23.5 33.0 27.9 2.88 119 
6 7 12 7 


What were teachers’ perceptions regarding the impact of 
secondary TDS’ coaching practices toward improving their 
instructional skills and increasing student achievement? 


Teachers’ perceptions of secondary TDS’ coaching practices 
were based on their responses to survey items on five indicators: 
Expertise and Credibility, Listening, Objectivity, Clear and 
Effective Communication, and Overall Impact. The findings are 
depicted in Table 7 through Table 11 (Appendix D, pp. 16-17). 
Results show the percentages of teachers who indicated that they 
“strongly agree” = “4”, “somewhat agree” = “3”, “somewhat 
disagree” = “2”, and “disagree” = “1” to survey items. Descriptive 
statistics, including means and standard deviations are presented. 

Table 7 (Appendix D, p. 16) shows teachers’ perceptions on 
the Expertise and Credibility indicator. An overwhelming majority 
of teachers responded that they “strongly agree” on 100% of the 
items. The findings ranged from 71.0% (the TDS has guided 
me towards relevant and meaningful professional development 
activities) to 83.8% (the TDS is knowledgeable about curriculum 
and instruction). Table 7a shows that agreement ratings on this 
indicator ranged from 3.59 to 3.80, revealing that teachers “more 
than somewhat agree” that TDS demonstrated expertise and 
credibility as a coach. 

Table 8 (Appendix D, p. 16) shows teachers’ perceptions on 
the Objectivity indicator. The vast majority of teachers responded 
that they “strongly agree” on 100% of the items. The findings 
ranged from 70.8% (the TDS is able to dialogue with me without 
projecting his/her own values and beliefs onto the conversation) 
to 75.5% (the TDS effectively pushes me to change/expand my 
practice while simultaneously acknowledging my successes). The 
agreement ratings ranged from 3.51 to 3.63 (Table 8a). In general, 
teachers revealed they “more than somewhat agree” that TDS 
displayed objectivity as a coach. 

Table 9 (Appendix D, p. 16) depicts teachers’ perceptions on 
the Listening indicator. The vast majority of teachers responded 
that they “strongly agree” on 100% of the items. The findings 
ranged from 74.9% (the TDS ensures that I am heard by my 
colleagues during group processes) to 82.0% (when we are 
having conversations, the TDS avoids distractions and devotes 
full attention to me at that particular time). The agreement ratings 
ranged from 3.64 to 3.76 (Table 9a, p. 8). Teachers revealed 
they “more than somewhat agree” that TDS exhibited listening 
behavior. 


Table 7a: Teachers coached by TDS, Expertise and Credibility agreement 


ratings, spring 2020 (n = 254) 


Agreement 
Expertise and Credibility Indicator Ratings S.D. 
u 6 
More than Somewhat Agree 
The TDS is knowledgeable about curriculum and instruc- 3.80 0.507 
tion. 
The TDS is aware of my classroom realities and adapts 3.63 0.743 
accordingly. 
The TDS acknowledges what I know while adding his/her 3.62 0.755 
own expertise to our discussions. 
I find the TDS’s experience relevant to my needs. 3.62 0.699 
The TDS has guided me towards relevant and meaningful 3.59 0.739 


professional development activities. 


Table 8a: Teachers coached by TDS, Objectivity agreement ratings, spring 


2020 (n = 254) 


Agreement 
Objectivity Indicator Ratings S.D. 
mn 0 
More than Somewhat Agree 
The TDS effectively pushes me to change/expand 3.63 0.745 
my practice while simultaneously acknowledging my 
successes. 
The TDS provides me with a balance of positive and 3.63 0.76 
constructive feedback. 
The TDS is able to dialogue with me without 3.56 0.806 
projecting his/her own values and beliefs onto the 
conversation. 
I feel that the TDS is my partner. 3.51 0.873 


Table 10 (Appendix D, p. 16) reflects teachers’ perceptions 
on the Clear and Effective Communication indicator. The vast 
majority of teachers responded that they “strongly agree” on the 
three items. The findings ranged from 75.6% (the TDS is open 
to feedback on his/her communication style and is willing to 
change practices in order to meet the needs of teachers and staff) 
to 78.6% (the TDS keeps professional conversations with me and 
other staff focused on the achievement of agreed-upon outcomes). 
The agreement ratings ranged from 3.59 to 3.68 (Table 10a, p. 
8). Teachers revealed they “more than somewhat agree” that TDS 
demonstrated clear and effective communication skills as a coach. 

Table 11 (Appendix D, p. 17) depicts teachers’ perceptions 
on the Overall Impact indicator. The vast majority of teachers 
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Table 9a: Teachers coached by TDS, Listening agreement ratings, spring 


2020 (n = 254) 


Table 11a: Teachers Coached by TDS, Overall Impact agreement ratings, 


spring 2020 (n = 254) 


Table 10a: Teachers coached by TDS, Clear and Effective Communication 


agreement ratings, spring 2020 (n = 254) 


Clear and Effective Communication Indicator Agreement 
Ratings S.D. 
mn o 
More than Somewhat Agree 
The TDS keeps professional conversations with me and 3.68 0.708 
other staff focused on the achievement of agreed-upon 
outcomes. 
The TDS provides me with clear, specific, and accurate 3.64 0.736 
feedback about classroom observations. 
The TDS is open to feedback on his/her commu- 3.59 0.831 


nication style and is willing to change practices in 
order to meet the needs of teachers and staff. 


responded that they “strongly agree” on the seven items. The 
findings ranged from 74.0% (the TDS has helped me develop 
my lessons to better meet the individual needs of my students) to 
78.0% (I feel the TDS provided high-quality support). Table lla 
shows agreement ratings on this indicator ranged from 3.59 to 3.63. 
In summary, teachers revealed they “more than somewhat agree” 
that TDS had a positive impact on their professional practice and 
helped them meet the individual needs of their students. 

Correlation analyses examined the strength of the 
relationships between the indicators used in the teacher survey 
(Table 12). There were large, positive associations evident 
in the data, with the largest association found among teachers’ 
perceptions of TDS’ objectivity when coaching and their overall 
impact on their instructional practices, r = .950, p < .001 (Table 
12). The second largest association was found among teachers’ 
perceptions of TDS listening behavior and their ability to deliver 
clear and effective communication, r= .944, p< .001. 

Finally, teachers were asked to provide their perceptions 
regarding how working with TDS as a coach helped them be 
more successful. A graphical depiction of responses are shown 
in Figure 6 (p. 9). Teachers’ comments were, mostly, positive. A 
sample of the comments follows. 


Listening Indicators Agreement Agreement 
Ratings S.D. Overall Impact Indicator Ratings S.D. 

More than Somewhat Agree More than Somewhat Agree 
When we are having conversations, the TDS avoids dis- 3.76 0.593 Coaching from TDS has made me more reflective about 3.63 0.747 
tractions and devotes full attention to me at that particular my practice. 
ne: Coaching from TDS has positively impacted my profes- 3.63 0.763 
The TDS regularly provides forums for me to express 3.63 0.777 sional practice. 
CONS EINE ne: ASI GMeSHIOnS: I feel the TDS provided high-quality support. 3.63 0.798 
The TDS provides safe opportunities for me to 3.63 0.76 The TDS has helped me implement the instruction- 3.62 0.776 
reflect honestly on my practice. al goals identified by the school. 
The TDS listens attentively to understand my 3.68 0.703 The TDS has helped me develop my lessons to 3.60 0.781 
aise better meet the individual needs of my students. 
The TDS is responsive to my questions and 3.69 0.735 The impact of the coaching from TDS on my 3.59 0.831 
ihe practice justifies the time I spent working with the 
The TDS is accessible. 3.75 0.59 coach. 
The TDS gives my concerns and ideas serious 3.62 0.822 Coaching from TDS has helped my grade-level 3.59 0.806 
consideration. team/department focus its work more effectively. 
The TDS ensures that I am heard by my colleagues 3.64 0.712 
during group processes. 
The TDS effectively listens to and synthesizes 3.64 0.77 Table 12: Pearson product-moment correlation between teacher in- 
what teachers and staff say. dicators 


Scale 1 2 3 4 5 


1. Expertise and Credibility - 


2. Listening 893** — - 

3. Objectivity 586** = .908**  - 

4. Clear/Effective Commun. soo2"* D19** — O44ee - 

5. Overall Impact 902"* .937** 950%" 938" = 
Mean 2.936 3.706 3.595 3.652 3.635 
S.D. 4834 5999 = 74257025 7296 


py <,001 (2-tailed) (N = 53) 


¢ “T feel more confident in my teaching and now have more 
knowledge of the content to better help my students.” 

« “Working with the TDS as a coach has given [me] the 
confidence, drive and energy to become a better teacher.” 

¢ “Tf I am uncertain about anything she takes the time to 
make sure that I fully understand before I attempt to get 
in front of my students to deliver the material.” 

e “She has helped me to prepare for my classes and make 
sure I have the materials I need to service my students.” 


What was the effect of the intensive secondary TDS coaching 
model on student performance in mathematics, science, 
reading, and social studies at targeted schools? 


Propensity score matching allowed for measurement of 
intensive secondary TDS treatment effects using non-randomized 
data, controlling for whether or not students were economically- 
disadvantaged, at risk, gifted/talented, male or female, receiving 
special education services, and their eighth-grade performance 
on content-related STAAR tests. Baseline characteristics (before 
matching) of TDS and non-TDS student groups are presented in 
Appendix E (Tables 13a—16b, pp. 18-19). Students at schools 
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Figure 6: Graphical depiction of teachers perceptions of how secondary 
TDS’ helped them be more successful teachers, spring 2020 


with secondary TDS were matched with students at schools that 
did not deliver intensive secondary TDS coaching practices. It 
should be noted that students at non-TDS intensive schools tended 
to have substantially lower percentages of students who were at 
risk of dropping out of school and much higher percentages of 
students who are gifted/talented. A total of 1,942 matches were 
found for students who completed the Algebra 1 DLA; 2,212 
matches for the Biology DLA; 3,161 matches for the ELA DLA, 
and 1,730 matches for the U.S. History DLA, taking into account 
the covariates. Findings related to the alignment of DLA with 
STAAR are discussed. Passing rates comparable to STAAR were 
derived by dividing the mean DLA score in the content area by the 
maximum score. (See Appendix F, p. 20 for information about 
the tests.) 

Figure 7 shows that the mean Algebra 1 DLA was lower 
for the TDS treatment group relative to the control group before 
matching (M = 12.66, S.D. = 5.261 vs. M = 12.82, S.D. 5.117). 
After matching, the mean Algebra 1 DLA exceeded the control 
group mean (M = 12.85, S.D. = 5.358 vs. M = 12.82, S.D. = 
5.117). On average, the mean Algebra I DLA score progressed 
closer toward the Meets Grade Level standard with intensive TDS 
treatment (46.9% to 47.6% passing). 

Figure 8 shows that the mean Biology DLA was much lower 
for the treatment group compared to the control group before 
matching (M = 14.68, S.D. = 5.758 vs. M = 16.34, S.D. 5.732). 
After matching, there was a reduction in the gaps between the 
Biology DLA treatment group and the control group (M = 15.204, 
S.D. = 5.923 vs. M = 15.485, S.D. = 5.585) from 1.67 points to 
.028 points. On average, the mean Biology DLA score moved 
closer toward the Meets Grade Level standard with intensive TDS 
treatment (48.9% to 50.7% passing). 

The mean ELA DLA was moderately lower for the treatment 
group relative to the control group before matching (M = 17.495, 
S.D. = 5.795 vs. M = 21.174, S.D. = 5.429) (Figure 9). After 
matching, there was a reduction in the gap between the ELA DLA 
treatment group and the control group (M = 18.361, S.D. = 5.730 
vs. M= 21.061, S.D. = 5.427), from 3.68 points to 2.70 points. On 
average, the mean ELA DLA score advanced toward the Meets 
Grade Level standard with intensive TDS treatment (56.4% to 


59.2% passing). 

The mean U.S. History DLA was slightly lower for the 
treatment group than the control group before matching (M = 
14.055, S.D. = 4.795 vs. M = 14.188, S.D. = 5.298) (Figure 10, p. 
10). After matching, the mean U.S. History DLA for the treatment 
group exceeded the mean score for the control group (M = 14.292, 
S.D. =4.748 vs. M= 13.901, S.D. =5.304). The difference between 
the groups was statistically significant (p < .05). Moreover, on 
average, the mean U.S. History DLA score progressed toward the 
Meets Grade Level standard with intensive TDS treatment (56.2% 
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Figure 7: Math propensity-score matching results using fall 


2019 Algebra 1 DLA as an outcome measure (Max. score = 27) 


Biology DLA 


ieee: 15.20 15.49 


Before Matching After Matching 


Mean Score 


= TDS Treatment Control 


Figure 8: Science propensity-score matching results using fall 
2019 Biology DLA as an outcome measure (Max. score = 30) 
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Figure 9: Reading propensity-score matching results using 


fall 2019 ELA DLA as an outcome measure (Max. score = 31) 
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Figure 10: Social studies propensity-score matching results using 
fall 2019 U.S. History DLA as an outcome measure (Max. score = 25) 


to 57.2% passing). 

In summary, the effects of intensive secondary TDS treatment 
led to mostly comparable or higher performance levels for students 
in schools with large proportions of at-risk students compared to 
schools with considerably less at-risk students based on students’ 
fall DLA Algebra 1, Biology, ELA, and U.S. History results. 
Consequently, secondary TDS treatment appeared to have benefits 
for students in struggling schools. 


Discussion 

Teacher coaching has emerged as a promising alternative 
to traditional professional development models to help teachers 
improve their instructional practices and to facilitate student 
achievement. Secondary Teacher Development Specialists have 
embraced a theory of action that is student-centered, collaborative, 
and that incorporates individualized, intensive, sustained, context- 
specific, and focused coaching along with instructional strategies 
and curricular materials to support literacy at each academic level. 
This program evaluation examined the perceptions of secondary 
TDS regarding their coaching competencies in key content areas. 
Moreover, the study assessed the extent secondary TDS coaching 
practices improved teachers’ instructional competencies to support 
student achievement at struggling schools in HISD. 

Pearson correlations revealed a strong, positive association 
between secondary TDS’ listening behavior and their ability to 
be objective and provide clear and effective communication to 
teachers. Teachers coached by secondary TDS, overwhelmingly, 
strongly agreed that they displayed expertise and credibility, 
listening, objectivity, and clear and effective communication skills 
as coaches. Moreover, teachers consistently noted that secondary 
TDS were knowledgeable about curriculum and _ instruction, 
pushed them to change and expand their practices, and helped 
them meet students’ individual needs. 

Propensity score matching allowed for measurement of 
intensive secondary TDS treatment effects (at least 80% to 90% of 
time spent coaching) at targeted schools using DLAs, controlling 
for students’ background and eighth-grade performance on 
content-related tests. The effects of secondary TDS treatment led 
to more comparable performance of at-risk students compared 
to students in schools with considerably less at-risk students in 
key content areas. Most notably, the mean U.S. History DLA 


was slightly lower for the TDS treatment group compared to the 
control group before matching, but significantly exceeded the 
control group’s mean score after matching. On average, the mean 
Algebra I, Biology, ELA, and U.S. History DLA scores advanced 
toward the Meets Grade Level standard with intensive secondary 
TDS treatment. 

There were several limitations to the study. Specifically, 
secondary TDS and teacher participation in surveys was voluntary 
and there were low DLA student participation rates. However, 
time was extended for survey completion and propensity-score 
matching techniques were used to select students who completed 
the assessments to establish treatment and control groups. The 
study mainly assessed high-school performance outcomes; 
however, middle-school outcomes were used as covariates for 
each content area. Future research may consider assessing the 
direct impact of TDS in middle-school environments. 

Research emphasizes that when practitioners experience new 
information as learners, the transfer of the new learning is evident 
in practices and routines. Leaders must provide opportunities 
for TDS to grow and develop by investing sufficient time and 
funding. Learning opportunities should focus on the specific 
needs of secondary TDS relative to specific content, pedagogy, 
and instructional coaching to maintain and accelerate positive 
student outcomes evident in this evaluation. 

Considering these findings, a true student-centered model 
should continue to underpin the work of HISD secondary TDS. 
Moreover, curriculum administrators should consider using 
these types of teacher coaching models to mitigate the social 
and economic barriers that students often face in high-risk 
school environments. Monitoring program implementation, 
test performance of students, and gathering feedback from 
stakeholders will help to ensure that the model is delivered by 
experts in the field as designed, continues to be useful to teachers, 
and has a positive impact on student achievement. 
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Appendix A 


Background chaaracteristics of the study samples, 2019-2020 


Secondary TDS Sample Teachers Coached by TDS Sample 
N % N % 
(53) (254) 
Content Areas 
English Language Arts 17 32.1 71 28.0 
Instructional Technology 4 a 2 0.8 
Literacy 6 11.3 0 0.0 
Math 7 13.2 47 18.5 
Science 12 22.6 67 26.4 
Social Studies 7 13.2 65 25.6 
Other - - 2 0.8 
Years as a Teacher 
1 - - 71 28.0 
2 - - 48 18.9 
3 - - 33 13.0 
4 = 102 40.2 
Source: Secondary TDS and teacher surveys 
41 secondary schools were represented in the data 


Appendix B 


Coaching Competencies Definition of Terms 


Clear and effective communication: Coaches are able to provide safe, structured opportunities for individual reflection; be transparent in 
their intention and approach, and thoughtful about the processes they utilize to do their work. Coaches should also maintain an objective pres- 
ence in group interactions; know and apply theories of group formation to maximize collaborative opportunities; and have a robust arsenal 
of tools and structures for moving groups towards their goals. 


Expertise and Credibility: coaches should have credible and relevant classroom experience; have knowledge beyond their classroom expe- 
rience, through research, reading, and professional development; anticipate realities of implementation in the classroom; and help teachers 
make connections and synthesize their experiences. Coaches should also understand and apply theories of organizational change and systems 
thinking, anticipate realities of implementation across multiple classrooms, identify and synthesize patterns of practice across a school, and 
be prepared to facilitate and/or coordinate professional development. 


Listening: Coaches act as a mirror that reflects the teacher’s plans and actions without judgment, analysis, or critique; draw out and help 
the teacher synthesie that information. Coaches should also maintain a balanced nd objective perspective in the face o onflict or controversy; 
validate multiple perspectives and approaches, while challenging staff to critically examine the impct of their classroom practices. 


Objectivity: Coaches have the ability to act and respond without judgment or critique, maintaining a balanced perspective in the coaching 
relationship. 


Adapted from: 
Insight Education Group. (2017). Coaching for change: 3 tools for assessing coaching competencies. Retrieved from http://www. insighted- 
ucationgroup.com/instructional-coaching-competencies-tools 
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Appendix C 


Table 2: Teacher Development Specialists’ Expertise and Credibility indicator responses, spring 2020 (n = 53) 


Tamcom- | Iam fairly Tam Tam Iam not at | Confidence Std. 
Expertise and Credibility Indicators pletely confident somewhat slightly all confi- Score Devia. 

confident | about this. confident confident dent about 

about this. about this. | about this. this. 

% % % % % u 0 

I am aware of the classroom realities at my site, and adapt my coaching accordingly. 714 15.1 75 - - 3.70 .607 
The teachers I work with find my experience relevant to their own needs. 64,2 22.6 11.3 1.9 - 3.49 775 
Ihave the credibility to help teachers reflect deeply on their own practices. 69.8 22.6 7.5 - - 3.62 627 
I share what I have learned in professional development with my staff. 83.0 11.3 5.7 - - 3.77 542 
I guide teachers and other colleagues towards relevant and meaningful professional 67.9 28.3 1.9 0.0 1.9 3.60 716 
development activities. 
I regularly observe in classrooms at my own site and at other sites so that I can 64.2 20.8 Sef 1.9 75 3.32 1.173 
identify instructional patterns and best practices. 
Ihave connected with my coaching mentor to get support in my current position. 54.7 13.2 11.3 3.8 17.0 2.85 1.537 
I bring my own knowledge to bear in conversations while acknowledging and 79.2 17.0 3.8 - - 3.13 S15: 
integrating the knowledge of others. 


Table 3: Teacher Development Specialists’ Listening indicator responses, spring 2020 (n = 53) 


lam com- Tam fairly Tam Tam slight- | Iam notat | Confidence Std. 
Listening Indicators pletely confident somewhat | ly confident all confi- Score Devia. 

confident about this. confident about this. | dent about 

about this. about this. this. 

% % % % % ul o 

I regularly provide a forum for teachers to express concerns and ask questions. 714 15.1 ef 1.9 - 3.68 .673 
I give teacher and staff concerns and ideas serious consideration. 83.0 17.0 - - - 3.83 379 
I am able to distinguish between concerns that simply need validation and those 69.8 24.5 3.7 - - 3.64 591 
that require action. 
I ask clarifying questions when appropriate to ensure that I understand what 79.2 17.0 3.8 - - 3315) 515 
teachers are saying. 
I provide safe opportunities for teachers and staff to reflect honestly on their 714 20.8 1.9 - - 3.75 ATT 
practice. 
When I have conversations with teachers, I avoid distractions and devote my 79.2 18.9 79.2 - - Sit 466 
full attention to their needs at that particular time. 
I effectively use facilitative processes that draw out reluctant participants. 49.1 35.8 9.4 5:7 - 3.28 .863 
I can ensure that all participants are heard by their colleagues during group 52.8 37.7 75 1.9 - 3.42 719 
processes. 
I can hear and process an issue from multiple perspectives (i.e., teacher, student, 77.4 17.0 5.7 - - a 568 
administrator, etc.). 
I provide opportunities for teachers to dialogue with each other, not just with 64.2 20.8 13.2 1.9 - 3.47 .799 
me. 
I effectively listen to and synthesize what teachers and staff say. dA 24.5 3.8 - - 3.68 547 


Note: This survey was adapted from Insight Education Group. (2017). Coaching for change: 3 tools for assessing coaching competencies. Retrieved from http://www. insighteduca- 
tiongroup.com/instructional-coaching-competencies-tools. 
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Appendix C (cont’d) 


Table 4: Teacher Development Specialists’ Objectivity indicator responses, spring 2020 (n = 53) 


Tam com- Iam fairly Tam lam slightly | Iam notat | Confidence Std. 
Objectivity Indicators pletely confident somewhat confident all confi- Score Devia. 

confident about this. confident about this. dent about 

about this. about this. this. 

% % % % % mM o 

I can listen to my collaborating teacher’s plans and observe his/her actions with- 60.4 34.0 5.7 - - 3.55 607 
out judgment, analysis, or critique. 
lask clarifying questions to draw out details and help teacers synthesize infor- 71.4 17.0 37 - - 3.72 568 
mation. 
Tam able to dialogue with teachers without projecting my own values and beliefs 60.4 26.4 11.3 1.9 - 3.45 774 
on the conversation. 
I make sure that I have all the relevant information about a situation before. 56.6 41.5 1.9 - - 3.55 539 
l actively solicit input from multiple stakeholders before taking action on an 67.9 20.8 9.4 - 1.9 3:53 .823 
issue. 
When facilitating a controversial conversation between more than one party, I am 58.5 35.8 5.7 - - 3.53 608 
able to remain neutral even when my personal beliefs might place me on one side 
of the dialogue. 
luse a variety of techniques to challenge staff to acknowledge their own assump- 41.5 41.5 9.4 Sar: 1.9 3.15 949 
tions. 
lam effective in pushing staff members to change/expand their practices while 54.7 35.8 Prt) 3.8 - 3.42 .770 
simultaneously acknowledging their successes. 
I balance feedback that I give to teachers and staff between positive and construc- 75.5 24.5 - - - 3.75 434 
tive. 


Table 5: Teacher Development Specialists’ Clear and Effective Communication survey responses, spring 2020 (n = 53) 


ITamcom- | Iam fairly Tam Tam Iamnotat | Confidence Std. 
Clear and Effective Communication Indicators pletely confident somewhat slightly all confident Score Devia. 

confident | about this. confident confident about this. 

about this. about this. | about this. 

% % % % % u o 

I provide clear, specific, and accurate feedback to teachers about classroom obser- 73.6 24.5 1.9 - - 3.12. 495 
vations. 
I meet face to face with teachers in a timely fashion following classroom observa- 71.7 20.8 5.7 1.9 - 3.62 686 
tions to debrief the observation and clarify written feedback. 
I model the process of articulating goals that will elicit specific outcomes. 64.2 30.2 3.8 1.9 - 3.57 665 
I keep professional conversations with teachers and staff focused on the achievement 83.0 17.0 - - - 3.83 379 
of agreed-upon outcomes. 
During conversations with teachers, I deliberately choose to use language that is 81.1 18.9 - - - 3.81 395 
clear, non-threatening, and objective. 
When appropriate, I share agendas and planning tools with teachers and staff prior 75:5 20.8 3.8 - - 3y12: .533 
to meetings. 
I model ways to communicate effectively. 73.6 24.5 1.9 - - 3:72 495 
I return phone calls and emails in a timely fashion (within 24 hours, when possible). (pe) 22.6 1.9 - - 3.74 486 
I am open to feedback on my communication style and am willing to change my 84.9 15.1 - - - 3.85 361 
practices in order to meet the needs of teachers and staff. 
I disseminate written information in a timely fashion. 77.4 22.6 - - - 3.77 423 
I use communication formats that are convenient for teachers and staff (i.e., bulletin 79.2 18.9 1.9 - - 3.77 466 
boards, email, paper memos). 
I provide clear processes and agendas for meetings and group conversations. 69.8 28.3 1.9 - - 3.68 510 


Note: This survey was adapted from Insight Education Group. (2017). Coaching for change: 3 tools for assessing coaching competencies. Retrieved from http://www. insighteduca- 


tiongroup.com/instructional-coaching-competencies-tools. 
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Appendix D 


Table 7: Teachers Coached by TDS Expertise and Credibility indicator agreement responses, spring 2020 (n = 254) 


Strongly Somewhat agree Somewhat Disagree Mean Std. Devia. 

Expertise and Credibility Indicators agree disagree 
% % % % mv o 

The TDS is aware of my classroom realities and adapts accordingly. 74.7 17.4 4.0 4.0 3.63 743, 
I find the TDS’ experience relevant to my needs. 71.7 22.0 3.1 3.1 3.62 699 
The TDS has guided me towards relevant and meaningful professional 71.0 20.2 3:6 3.2. 3:39 .739 
development activities. 
The TDS acknowledges what I know while adding his/her own expertise to 73.9 18.2 3.6 43 3.62 135 
our discussions. 
The TDS is knowledgeable about curriculum and instruction. 83.8 13.0 24 0.8 3.80 507 


Table 8: Teachers Coached by TDS Objectivity indicator agreement responses, spring 2020 (n = 254) 


Strongly Somewhat agree Somewhat Disagree Mean Std. Devia. 

Objectivity Indicators agree disagree 
% % % % u 0 

I feel that the TDS is my partner. 70.9 15.5 7.6 6.0 3:31 873 
The TDS is able to dialogue with me without projecting his/her own values 70.8 18.8 5.6 4.8 3.56 806 
and beliefs onto the conversation. 
The TDS effectively pushes me to change/expand my practice while simulta- ore 16.5 4.0 4.0 3.63 745 
neously acknowledging my successes. 
The TDS provides me with a balance of positive and constructive feedback. 75.4 16.3 4.0 4.4 3.63 760 


Table 9: Teachers Coached by TDS Listening indicator agreement responses, spring 2020 (n = 254) 


Strongly Somewhat agree Somewhat Disagree Mean Std. Devia. 

Listening Indicators agree disagree 
% % % % u o 

The TDS is responsive to my questions and concerns. 81.0 12.3 2.0 4.8 3.69 735 
The TDS is accessible. 81.0 15.1 2.0 2.0 3.75 590 
The TDS listens attentively to understand my needs. 78.7 14.6 3.2 3.6 3.68 .703 
The TDS regularly provides forums for me to express concerns and ask 76.4 14.8 4.0 4.8 3.63 777 
questions. 
The TDS gives my concerns and ideas serious consideration. 771.8 12.3 4.0 6.0 3.62 822 
The TDS provides safe opportunities for me to reflect honestly on my 76.5 14.3 5.2 4.0 3.63 760 
practice. 
When we are having conversations, the TDS avoids distractions and devotes 82.0 14.4 11,2) 2.4 3,76 593 
full attention to me at that particular time. 
The TDS ensures that I am heard by my colleagues during group processes. 74.9 18.0 3.8 3.3 3.64 712 
The TDS effectively listens to and synthesizes what teachers and staff say. 715 14.1 3.6 4.8 3.64 .770 


Table 10: Teachers Coached by TDS Clear and Effective Communication indicator agreement responses, spring 2020 (n = 254) 


Strongly Somewhat agree Somewhat Disagree Mean Std. Devia. 

Clear and Effective Communication Indicators agree disagree 
% % % % u o 

The TDS provides me with clear, specific, and accurate feedback about class- 75.7 15.4 5.7 3.2 3.64 .736 
room observations. 
The TDS keeps professional conversations with me and other staff focused on 78.6 14.5 3.2. 3.6 3.68 708 
the achievement of agreed-upon outcomes. 
The TDS is open to feedback on his/her communication style and is willing to 75.6 14.2 4.1 6.1 3.59 831 
change practices in order to meet the needs of teachers and staff. 


Note: This survey was adapted from Insight Education Group. (2017). Coaching for change: 3 tools for assessing coaching competencies. Retrieved from http://www.insighteduca- 
tiongroup.com/instructional-coaching-competencies-tools. 
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Appendix D (cont’’d) 


Table 11: Teachers Coached by TDS Overall Impact indicator agreement responses, spring 2020 (n = 254) 


Strongly Somewhat agree Somewhat Disagree Mean Std. Devia. 

Overall Impact Indicators agree disagree 
% % % % vy o 

Coaching from TDS has made me more reflective about my practice. 75.0 16.9 4.0 4.0 3.63 TAT 
Coaching from TDS has positively impacted my professional practice. 75.5 16.1 4.0 44 3.63 763 
The impact of the coaching from TDS on my practice justifies the time I spent 15.3 13.8 5.3 5.7 3.59 831 
working with the coach. 
The TDS has helped me implement the instructional goals identified by the school. 75.4 15.3 4.8 4.4 3.62 776 
The TDS has helped me develop my lessons to better meet the individual needs of 74.0 16.4 5.2 4.4 3.60 .781 
my students. 
I feel the TDS provided high-quality support. 78.0 11.6 5.6 4.8 3.63 798 
Coaching from TDS has helped my grade-level team/department focus its work 74.8 14.6 3.7 4.9 3.59 .806 
more effectively. 


Note: This survey was adapted from Insight Education Group. (2017). Coaching for change: 3 tools for assessing coaching competencies. Retrieved from http://www. insighteduca- 
tiongroup.com/instructional-coaching-competencies-tools. 
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Table 13a: Math propensity score matching - background covariates 


Appendix E 


Before Matching After Matching 
N Eco. Sp. At G/T Male Female N Eco. Sp. At G/T Male Female 
Disadv. Ed. Risk Disadv. | Ed. | Risk 
% % % % % % % % % % % % 
TDS Treatment 3594 93.6 10.8 87.2 6.9 52.2 47.8 1942 89.2 10.1 | 65.1 | 25.4 52.8 47.2 
Group 
Control Group 1942 76.0 7.6 65.1 25.4 45.6 54.4 1942 76.0 7.6 | 82.0 | 10.9 45.6 54.4 


Table 13b: Math propensity score matching - assessment covariate and outcome measure 


8th Grade Math Covariate (Spring 2019) 


Algebra I DLA (Fall 2019) Outcome Measure 


n Mean Std. Mean t df p n Mean Std. Mean df p. 
Before Matching Scale Devia. Diff Score | Devia. Diff 
Score 
TDS Treatment 3591 | 1642.55 | 128.941 | 64.874 | 16.935 5531 000 3594 12.66 5.261 .163 1.108 5534 .268 
Group 
Control Group 1942 | 1707.42 | 148.165 1942 12.82 S117 


After Matching n Mean Std. Mean t df p n Mean Std. Mean df p 
Scale Devia. Diff Score | Devia. Diff 
Score 
TDS Treatment 1942 | 1651.88 | 133.968 | 55.542 | 12.253 3882 .000 1942 12.85 5.358 .033 196 3882 845 
Group 
Control Group 1942 | 1707.42 | 148.165 1942 12.82 5.117 


Table 14a: Science propensity score matching - background covariates 


Before Matching After Matching 
N Eco. Sp. At G/T Male Female N Eco. Sp. At G/T Male Female 
Disadv. Ed. Risk Disadv. | Ed. | Risk 
% % % % % % % % % % % % 
TDS Treatment 3157 93.1 6.7 43.5 11.2 44.2 55.8 2212 90.2 6.1 | 72.6 | 15.7 51.7 48.3 
Group 
Control Group 2867 65.0 3.5 80.0 46.6 50.6 49.4 2212 76.2 4.2 | 53.1 | 35.1 44.9 55.1 


8th Grade Science Covariate (Spring 2019) 


Table 14b. Science propensity score matching - assessment covariate and outcome measure 


Biology DLA (Fall 2019) Outcome Measure 


n Mean Std. Mean t df p n Mean Std. Mean t df p. 
Before Matching Scale Devia. Diff Score Devia. Diff 
Score 
TDS Treatment 3156 | 3712.75 | 507.389 | 494.282 | 34.144 | 6020 .000 3157 14.677 | 5.7580 | 1.6646 | 11.230 | 6022 | .000 
Group 
Control Group 2866 | 4207.04 | 614.747 2867 16.342 | 5.7318 


After Matching n Mean Std. Mean t df p n Mean Std. Mean t df p 
Scale Devia. Diff Score Devia. Diff 
Score 
TDS Treatment 2212 | 3744.90 | 525.224 | 295.577 | 17.983 | 4422 000 2212 15.204 | 5.9229 .2807 1.622 | 4422 | .105 
Group 
Control Group 2212 | 4070.48 | 567.195 2212 15.485 | 5.5850 


Appendix E (cont’d) 


Table 15a: Reading propensity score matching - background covariates 


Before Matching After Matching 
N Eco. Sp. At G/T Male Female N Eco. Sp. At G/T Male Female 
Disadv. Ed. Risk Disadv. Ed. | Risk 
% % % % % % % % % % % % 
TDS Treatment 4433 93.0 9.8 80.6 12.9 51.4 48.6 3161 91.0 8.2 | 74.5 | 17.7 55.0 45.0 
Group 
Control Group 3161 714 4.9 51.5 40.5 44.2 55.8 3161 714 4.9 | 51.5 | 40.5 44.2 55.8 


8th Grade Reading Covariate (Spring 2019) 


Table 15b: Reading propensity score matching - assessment covariate and outcome measure 


ELA DLA (Fall 2019) Outcome Measure 


Mean 


Std. Mean t 


df 


n p n Mean Std. Mean t df p. 
Before Matching Scale Devia. Diff Score Devia. Diff 
Score 
TDS Treatment 4433 | 1618.32 | 130.714 | 106.208 | 34.694 | 7688 .000 4433 17.4948 | 5.7953 | 3.6793 | 28.251 | 7688 | .000 
Group 
Control Group 3257 | 1724.53 | 135.228 3257 21.1741 5.429 


After Matching n Mean Std. Mean t df Pp n Mean Std. Mean t df p. 
Scale Devia. Diff Score Devia. Diff 
Score 
TDS Treatment 3161 | 1647.49 | 126.844 | 72.019 | 22.021 6320 .000 3161 18.3614 | 5.72956 | 2.699 19.230 | 6320 | .000 
Group 
Control Group 3161 | 1719.51 | 133.115 3161 21.0607 | 5.42738 


Table 16a: Social studies propensity score matching - background covariates 


Before Matching After Matching 
N Eco. Sp. At G/T Male Female N Eco. Sp. At G/T Male Female 
Disadv. Ed. Risk Disadv. | Ed. | Risk 
% % % % % % % % % % % % 
TDS Treatment 2110 92.8 8.7 88.0 5.6 54.4 45.6 1730 91.4 7A 85.4 6.8 51.4 48.6 
Group 
Control Group 2008 71.8 4.0 62.3 29.2 48.9 51.1 1730 77.6 45 | 68.1 | 21.8 49.7 50.3 


Table 16b: Social studies propensity score matching - assessment covariate and outcome measure 


8th Grade Social Studies Covariate (Spring 2017) U.S. History DLA (Fall 2019) Outcome Measure 
n Mean Std. Mean t df p n Mean Std. Mean t df p. 
Before Matching Scale Devia. Diff Score Devia. Diff 
Score 
TDS Treatment 2110 | 3428.69 | 359.389 | 349.336 | 25.145 | 4116 000 2110 14.0548 | 4.7945 1332 847 4116 | .397 
Group 
Control Group 2008 | 3778.02 | 492.938 2008 14.1880 | 5.2980 


After Matching n Mean Std. Mean t df p n Mean Std. Mean t df p. 
Scale Devia. Diff Score Devia. Diff 
Score 
TDS Treatment 1730 | 3475.45 | 406.669 | 233.156 | 15.605 | 3458 .000 1730 14.2918 | 4.7481 3865 2.258 | 3458 | .024 
Group 
Control Group 1730 | 3708.61 | 469.991 1730 13.9053 | 5.3042 


Appendix F 


DLA and STAAR Alignment (Source: Houston Independent School District, Student Assessment Department (personal com- 
munication, 1/8/2020)) 


Approaches | Meets | Masters Approaches | Meets Masters 
% % % % % % 
ELA DLA 60 69 88 Biology DLA 39 61 83 
Algebra I DLA 39 63 78 U.S. History DLA 44 65 81 


State of Texas Assessments of Academic Readiness (STAAR®) Performance Labels and Policy Definitions 


Masters Grade Level* 

Performance in this category indicates that students are expected to succeed in the next grade or course with little or no academ- 
ic intervention. Students in this category demonstrate the ability to think critically and apply the assessed knowledge and skills 
in varied contexts, both familiar and unfamiliar. 

* For Algebra II and English III, this level of performance also indicates students are well prepared for postsecondary success. 


Meets Grade Level** 

Performance in this category indicates that students have a high likelihood of success in the next grade or course but may still 
need some short-term, targeted academic intervention. Students in this category generally demonstrate the ability to think crit- 
ically and apply the assessed knowledge and skills in familiar contexts. 

** For Algebra II and English III, this level of performance also indicates students are sufficiently prepared for postsecondary 
success. 


Approaches Grade Level 

Performance in this category indicates that students are likely to succeed in the next grade or course with targeted academic 
intervention. Students in this category generally demonstrate the ability to apply the assessed knowledge and skills in familiar 
contexts. 


Did Not Meet Grade Level 
Performance in this category indicates that students are unlikely to succeed in the next grade or course without significant, on- 
going academic intervention. Students in this category do not demonstrate a sufficient understanding of the assessed knowledge 


and skills. 


Citation: Texas Education Agency. (2017). Student Assessment Division. State of Texas Assessments of Academic Readiness 
(STAAR®) Performance Labels and Policy Definitions. Retrieved from https://tea.texas.gov/sites/default/files/STAAR_Perfor- 
mance Labels and Policy _Definitions.pdf 
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